This invention relates generally to the field of digital microscopes, and more specifically to digital microscope systems and methods for the rapid scanning of all or portions of a specimen on a microscope slide.
Since the development of the first working optical microscopes by Leeuwenhoek in the late 17th century, magnified images have been used in many different areas of scientific research. Improvements in lenses in the 18th and 19th centuries greatly improved the performance of conventional compound (i.e., multi-lens) microscopes in producing accurate images. The development of the electron microscope in the 20th century allowed scientists to obtain images of structures far smaller than those capable of viewing with optical microscopes. However, the use of optical microscopes at magnifications less than 100× remains highly important in many fields including, without limitation, botany, microbiology, geology, and medicine.
The utility of conventional compound optical microscopes for examining specimens mounted on glass slides is compromised because the field of view (FOV)—the portion of the specimen actually visible to the user at the eyepiece at any given time—becomes smaller as magnification increases. There is an inverse relationship between the optical magnification used to view the specimen and how much of the overall slide may be seen in the eyepiece FOV.
This becomes problematic when a user views the entire slide specimen, or a large part of it, under low magnification to identify a target area (also referred to as a region of interest or ROI) for viewing at high magnification. In optical microscopes, the user must switch objective lenses to a higher power and reacquire a (smaller) portion of the desired area at the higher magnification. Because the FOV at the higher magnification is so small, and because there is no cross-reference to identify the location of the new, higher magnification FOV within the overall slide specimen, the user may frequently miss portions of the ROI, be unsure whether the image acquired at the higher power is actually part of the ROI identified at the lower magnification, or be unsure of where the higher magnification field is located within the larger ROI. Essentially, the user's location within the “forest” of the overall slide becomes lost when focusing on the “trees” of a specific location at higher magnification.
If the user is a pathologist scanning for cancerous cells, in a tissue sample of a patient, for example, this could result in a missed diagnosis and obvious risk to patient health. In different contexts, the user may miss other desired structures, such as a particular cell wall region in a botanical sample, a particular group of microbial cells, specific crystal structures in a geological sample, or an area of high white blood cell counts in a blood sample.
In the last twenty-five years, the use of automated microscopes to generate digital images for examining microscope slides has become increasingly common. A class of microscopes using a combination of optical and electronic image acquisition and processing techniques, known as whole slide imaging (WSI) microscopes, has seen extensive growth. WSI microscopes are automated microscopes that use a camera with a digital image sensor (DIS) to capture a series of optically magnified (e.g., 20×, 40×, 60×, or 80×) digital images of adjacent, very small portions (each of which is usually referred to as a “field” or “tile”) of a target area or region of interest (ROI) of a microscope slide specimen. Because WSI microscope images are intended to be displayed on a computer or television screen or monitor, WSI microscopes typically lack the ocular or eyepiece lens in a standard compound microscope. Thus, magnification of the slide specimen is usually provided by a single objective lens (which may include a relay lens for infinity-focused objectives) in optical communication with the DIS. Stated differently, the optical path of a WSI microscope typically replaces the ocular lens with a digital image sensor (DIS).
To create a large, high-resolution image, each field image in the ROI slightly overlaps its adjacent field images, so that the field images collectively cover the entire target area/ROI. The digital field images or tiles scanned by the DIS camera are combined by computer software into a single, large magnified image file of the full ROI, which may in some instances comprise the entire slide. A variety of software algorithms may be used to combine the field images into the composite, magnified ROI image, which is similar to a panoramic image assembled by software from a series of overlapping images on non-microscopic digital cameras. In addition, a series of images at various lesser magnifications of the single, fully magnified ROI image may be created by image processing algorithms to digitally reduce the detail level in the fully magnified ROI, which provides the user with an artificial, digitally created “zoom in/zoom out” ability when viewing the complete ROI image comprising multiple field images.
WSI microscopes also help address the FOV problem noted above by providing an overview image of the entire specimen area of the slide at low (or zero) magnification in a first (overview image) window on a computer screen. The overview image may be taken by an overview camera at low or zero optical magnification and displayed as a thumbnail image in the overview window. With the overview image as a guide, the user may designate one or more target (ROI) areas for higher magnification through the objective lens(es) of the microscope using a pointing device such as a mouse, touchscreen, touchpad, etc.
In some embodiments, the overview image may be digitally enlarged, wherein the overview image is simply magnified by digitally adding additional pixels to create a larger image. The additional pixels, however, add no additional image detail. In this instance, the image appears large on the screen, but the image resolution remains the same. The digitally enlarged (low quality) overview image may assist the user in designating the one or more target or ROI areas for higher magnification and resolution.
In one mode of operation (“browse mode” or “live view mode”), the user may view or browse any area of the specimen at the magnification provided by the objective lens (e.g., 20×, 40×, 60×, etc.). A second (browse mode) window, usually larger than the first (overview image) window, may be displayed on the computer screen or monitor showing the field of view currently received by the digital image sensor from the objective lens of the WSI microscope. An overlaid indication (e.g., a box, crosshairs, or other highlighting) on the overview image is provided to visually designate where the current FOV image of the objective lens (i.e., the image in the browse window) is located within the overview image.
The browse/live view mode thus maps the current FOV of the objective lens back to the overview image of the specimen. In doing so, the user is provided with a visual indication of which “trees” (browse window image) within the slide “forest” (the overview) are being viewed at high magnification at any given time. In WSI microscopes at higher magnifications (e.g., 40× or 60×), the FOV of the objective lens may be so small that visualizing it within the overview image is enhanced by digitally enlarging the overview image in the overview image window, so that the objective lens FOV can be seen as a box or area on the overview image rather than a single point.
In another mode of operation (sometimes referred to as “scan mode,” “ROI mode,” or “zoom mode”), a portion of the ROI image, obtained by combining numerous individual field images or tiles, may be displayed in a third (scan or ROI mode) window on a monitor or display screen. As previously noted, a target area or ROI may be designed by the user using, e.g., a mouse or touchscreen. The user may then instruct the WSI microscope to scan the ROI in very small portions at high magnification to create a series of overlapping field images, which are then combined into a single, high resolution ROI image, which may be displayed in the third (scan or ROI) window.
The scan mode window may be shown on the monitor in addition to, or instead of, the second (browse mode) window. A small thumbnail image of the ROI may be displayed in a region of the third window (or in a separate fourth window), and a box or highlighted area within the ROI thumbnail image may show which portion of the entire ROI is being viewed in the scan mode window on the screen at any moment. Thus, the larger magnified image in ROI mode is mapped back to a thumbnail image of the entire ROI, similar to the way the browse mode (live or real-time) image from the objective lens is mapped back to the overview image in browse mode.
In some systems, the relative sizes or positions on the computer monitor or display screen of the first (overview) window, second (browse mode) window, and the third (scan or ROI image) may be determined by the user with a mouse or touchpad, allowing a customized viewing screen. In some embodiments, the user may view the overview (first) window and may toggle between the browse (second) window and the ROI image (third) window in the largest portion of the computer monitor or display.
While the overview, browse image, and ROI images have been described as being displayed in separate windows, in some embodiments the images may be displayed without using windows, and specific screen regions or areas may be used to provide different images, or the overview image may be displayed in, e.g., a corner of the browse or ROI images.
WSI microscopes having the foregoing functions and features are described in, e.g., U.S. Pat. Nos. 6,101,265 and 6,711,283, each of which is hereby incorporated by reference in its entirety. A compact WSI microscope having similar features to those described is available from Microscopes International, LLC (Plano, Tex.) with a 20×, 40× or 60× magnification as the uScope MXII microscope.
Although there are significant variations among commercially available systems, a WSI microscope typically includes a movable stage that holds the microscope slide. In some systems, the stage is motorized to move at a constant speed, and digital field images are taken at time intervals synchronized to the stage speed to obtain field images for combination into the high-resolution ROI image. These WSI systems may be referred to as moving image acquisition (MIA) systems. In other systems, the stage is motorized to move rapidly to a series of fixed positions from which the field images of the ROI are captured and subsequently combined. These WSI systems are referred to as fixed image acquisition (FIA) systems, because the image is obtained with the stage in a fixed (i.e., stationary) position. In both MIA and FIA systems, one or more motors are typically provided to move the stage in and out (X axis) and left and right (Y axis).
WSI microscopes also include an illumination system providing light to the slide stage, and an objective lens to magnify the light from the slide specimen and focus it on a digital image sensor (DIS) element in a digital camera. Focusing is typically provided by making one or more of the stage and the camera/objective lens structure (the camera and objective lens in WSI microscopes are typically coupled to a tube to maintain a fixed distance therebetween) movable by a motor (Z axis) capable of finely controlled, small movements on an axis generally perpendicular to the slide stage. This allows structures at different depths within the specimen to be captured in proper focus.
Digital image sensor cameras for WSI microscopes typically involve a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) image sensor as the DIS element. In some WSI systems, the DIS is incorporated into a static (snapshot) camera that captures field/tile images and combines them into a single, high-definition ROI image. In other systems, the DIS is incorporated into a video camera that outputs a stream of video images, from which frames may be captured as still image fields/tiles. Recent trends in digital photography, particularly digital video cameras, have seen a migration toward CMOS digital image sensors, which are most cost-effective than CCD sensors.
For video cameras, each distinct image that is output by the DIS element is known as a frame, and the frame output rates for such video cameras are measured in frames per second (FPS) with typical rates between 10 and 90 FPS. Thus, at a frame rate of 30 FPS (typical for many video cameras), each new image in the video stream is created every 33.33 milliseconds (mSec), which is referred to as the frame time (FT). Similarly, at 50 FPS, the images are captured every 20 mSec. For FIA systems, the images are obtained while the stage is not moving, so any video frame captured while the stage is stationary may be used as the field image or tile.
Once all the field images comprising a ROI have been captured, image combination algorithms are used to combine the field images into a single, high-definition ROI image. Although a variety of image combination algorithms are used, one class of algorithms known as pattern matching algorithms operates by mathematically aligning the edges of adjacent field images until they overlap, and then combining the images at the overlapping region.
A significant limitation associated with WSI microscopes is the time required to take and compile the field images or tiles for combination into the single high definition ROI image. To obtain a ROI image using either a MIA or FIA system, the WSI microscope objective is moved over the ROI, and magnified images are taken of very small areas either while the stage is moving (MIA systems) or fixed/stationary (FIA systems). The X-axis and Y-axis (stage movement) and Z-axis (focus) motors in the WSI microscope must be synchronized with the camera to properly capture digital field images.
In FIA systems, each X-axis, Y-axis, and Z-axis movement to a new stationary position takes a certain time (move time or MT) to occur, and causes a system vibration that requires a certain damping time (Settle Time or ST) to elapse before a new image may be taken without blurring. Thus, Frame Times are limited not only by the move time in moving the camera from a first image position to a second image position, but also by the settle time necessary to resolve the vibrations following the move.
Furthermore, after the settle time, any partially completed video frame output from the camera must be completed before the next video frame can be captured (frame completion time or FCT).
For some field images, called exhaustive focus fields (EFF), multiple images at the same stage position (i.e., X-axis and Y-axis location) are taken at different focus points (Z-axis positions). This involves a Z-axis focusing movement of either 1) the slide stage, or 2) the light path/tube containing the DIS and the objective lens, plus an additional settle time (ST) for the camera vibration associated with the Z-axis movement to dampen out, and a FCT period to complete the video frame output occurring when the Z-axis settle time elapses.
The entire process of completing a single frame for compilation into the ROI image includes the following steps:
1. move the slide stage to a desired (X, Y) location relative to the objective lens (MT);
2. wait for the vibration from the stage motion to dampen out (ST);
3. wait for the current partially completed video frame to finish (FCT);
4. capture the next complete video frame output from the camera;
5. change the focus position (Z axis location) of the objective at the same X, Y location (optional step for multiple images at the same field position to obtain the best focus);
6. wait for the focus (Z-axis) movement to dampen out (optional step for multiple images at the same field position to obtain the best focus);
7. wait for the current partially completed frame to finish (optional step for multiple images at the same field position to obtain the best focus);
8. capture the next complete image output from the camera (optional step for multiple images at the same field position to obtain the best focus);
9. repeat steps 5-8 for multiple images until all images at the same field position are obtained; the multiple images at the same location are referred to as a Z stack (optional step for multiple images at the same field position to obtain the best focus);
10. repeat steps 1-4 (for single-focus images) or 1-9 (for exhaustive focus fields) until all field images for the ROI have been scanned.
Ideally, all movements (X-axis, Y-axis, and Z-axis) would occur instantaneously, and with no settle time. This would allow each field image captured from the camera's video stream to have a valid, usable image. At 50 FPS, the camera would produce 50 field images each second.
In reality, the settle time for the stage (or objective) depends in part on the mass that moves. Thus, a movement of a heavier element (e.g., a DIS/objective lens tube (Z-axis movement), or a stage move (X-axis or Y-axis move) with the mounted slide) takes longer to settle than a lighter one. For a 50 FPS camera, if the total move time+settle time+frame completion time were less than 20 mSec (the time necessary for the camera to produce a full video image frame), then it would be possible to capture one video image frame and skip one frame while waiting for the move and settle times to lapse. The frame occurring immediately after the skipped frame could be captured and used, and the next movement could then occur, followed by a skipped frame, etc. This would effectively halve the camera frame rate from 50 to 25 images per second.
In most instances, however the move time and settle time exceed one frame time. In such cases, it is necessary to wait for additional whole frame(s) until the MT, ST, and FCT have elapsed. So, at 50 FPS if we ignore 2 frames, the effective FPS is 50÷3=16.67 FPS. If we ignore 3 frames, the effective FPS is 50÷4=12.5 FPS, and so on. Thus, if a Z stack is comprised of 25 images, and 1 frame must be ignored for move time+settle time+frame completion time for each field image of the Z stack, it takes 1.0 seconds to capture all 25 images from the camera.
There is a need for improved scan times and as to minimize one or more of the components of the Field Image Scan Time (e.g., one or more of FT, MT, ST, or FDT) to provide faster ROI images at high definition.